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METHODS AND APPARATUS FOR ERROR than I pictures through the use of motion compensation, and 

CONCEALMENT UTILIZING TEMPORAL also serve as a reference for B pictures and future P pictures. 

DOMAIN MOTION VECTOR ESTIMATION B ("bidirectional") pictures are fields or frames that use 

the most closest (with respect to display order) past and 

BACKGROUND OF THE INVENTION 5 future I or P picture as a reference, resulting in bidirectional 

1. Field of the Invention prediction. B pictures provide the most compression and 

-n. * • 11 * • j j- increase signal to noise ratio by averaging two pictures. 

The present invention relates generally to video encoding o L * n j « • * .1 5 j j • 

j j j' j • 1 . * . 0 Such I, P and B pictures are more thoroughly described m 

and decoding and, m parncular, to meihods and apparatus ' ? fj 

for error concealment in video encoding and decoding. in z, *. , ' " ' " 7 ' , \ L L 

^ . 10 Corporation and said U.S. Patents are incorporated herein by 

2. Descnption of the Related Art reference 

Advances in audio and video compression and decom- A of p i ct ures ("GOP") is a series of one or more 

pression techniques, together with very large scale integra- coc jcd pictures which assist in random accessing and editing, 

tion technology, have enabled the creation of new capabili- A G0P value ^ configurable during the encoding process, 

ties and markets. These include the storage of digital audio 15 since thc t pictures are closer together, the smaller the GOP 

and video in computers and on small optical discs as well as va i ue , the better the response to movement. The level of 

the transmission of digital audio and video signals from compression is, however, lower. 

direct broadcast satellites. In a oodcd bitstream ^ a GOp must sUrt ^ an j picture 

Such advances were made possible, in part, by interna- an d may be foUowed by any number of I, P or B pictures in 

tional standards which provide compatibility between dif- 20 any or d er . In display order, a GOP must start with an I or B 

ferent approaches to compression and decompression. One p i c ture and end with an I or P picture. Thus, the smallest 

such standard is known as "JPEG," for Joint Photographic GO p size is a single I picture, with the largest size being 

Expert Group. A later developed standard is known as unlimited. 

"MPEG 1 » This was the first set of standards agreed to by In detailj mQ x iUustrates a simplified block 
the Moving ^cturesE^ert Group. Yet another standard is * d{ of M MPEG2 encoder 100. A video stream con- 
known as ITU-T H.261 which is a video compression sistin of macroblock infection and motion compensation 
standard particularly useful for video teleconferenang. mforrnation k provided to bot h a discrete cosine transformer 
Although each standard is designed for a specific 102 and a motion vector generator 104. Each 8x8 block (of 
application, all of the standards have much m common pM& QI ^ tcrms) fa processed by the discrete 

MPEG1 was designed for storing and distributing audio transformer 102 to generate an 8x8 block of horizontal and 

and motion video, with emphasis on video quality. Its vertical frequency coefficients. The quantizer 106 quantizes 

features include random access, fast forward and reverse the 8x8 block of frequency-domain error coefficients, 

playback. MPEG1 serves as the basis for video compact thereby limiting the number of allowed values, 

disks and for many video games. The original channel mgher frcquencics are usually quantized more coarsely 

bandwidth and image resolution for MPEG1 were estab- than low frequencies, taking advantage of the human per- 

hshed based lupon the recording .media then available The ception of quan tization error. This results in many 

goal of MPEG1 was the reproduction of recorded digital f reqU encyKiomain error coefficients being.zero, especially at 

audio and video using a 12 centimeter diameter optical disc higher frequencies. 

with a bit rate of 1.416 Mbps, 1,15 Mbps of which are ^™ rt f 1ft £ • w , . n • rr 

„ , . , r r 40 I ne output or quantizer 106 is processed by a zigzag 

a oca e 0 vi eo. scanner 108, which, starting with DC components, generates 

Hie compressed bit streams generated under the MPEG1 a stream of quantized frequency coefficients arranged 

standard implicitly define the decompression algorithms to in order of mcr easing frequency. This produces long runs of 

be used for such bit streams. The compression algorithms, consecutive zero coefficients, which are sent to the variable 

however, can vary within the specifications of the MPEG 1 4$ length encoder 110 

standard, thereby allowing the possibility of a proprietary ^ ^ stfeam of tized freque ncy-domain error 

advantage in regard to the generation of compressed bit coefficients is first run-length encoded by the variable length 

streams. encoder 110, In the run-length encoding process, the linear 

A later developed standard known as "MPEG2" extends str eam of quantized frequency-domain error coefficients is 

the basic concepts of MPEG1 to cover a wider range of 50 converted into a series of run-amplitude (or run-level) pairs, 

applications. Although the primary application of the Each pair indicates the number of zero coefficients and the 

MPEG2 standards is the all digital transmission of amplitude of the non-zero coefficient which ends the run. 

broadcast-quality video at bit rates of 4 Mbps to 9 Mbps it For cxamplc> assume a string of crror coefficients as 

appears that the MPEG2 standard may also be useful for follows* 

other applications, such as the storage of full length motion 55 (1) 0riginal error coefficienls: 000060000038 

pictures on Digital Video Disk ("DVD") optical discs, with Therefore, when this string of error coefficients is variable 

resolution at least as good as that presently provided by 12 length encodedj acC ording to the encoding rules described 

inch diameter laser discs. above> the following encoded run-level pairs are obtained: 

The MPEG2 standard relies upon three types of coded (2) Encoded run-level pairs: (4,6) (5,3) (0,8) 
pictures. I ("intra") pictures are fields or frames coded as a 60 Of course, as the number of zero coefficients is increased, 
stand-alone still image. Such I pictures allow random access the error coefficient data will be more effectively corn- 
points within a video stream. As such, I pictures should pressed by this variable length encoding, 
occur about two times per second. I pictures should also be After the variable length encoder 110 encodes the run- 
used where scene cuts (such as in a motion picture) occur. level pairs, it then Huffman encodes the run-level pairs. In 

P ("predicted") pictures are fields or frames coded relative 65 the Huffman encoding, the run-level pairs are coded differ- 
to the nearest previous I or P picture, resulting in forward ently depending upon whether the run-level pair is included 
prediction processing. P pictures allow more compression in a list of commonly -occurring run-level pairs. If the 



06/01/2003, EAST Version: 1.03.0002 



6,078,616 

3 4 

run-level pair being Huffman encoded is on the list of created by the generated data (generally at the macroblock 

commonly-occurring pairs, then it will be encoded into a level) are not likely to be perceived by a viewer of a video 

predetermined variable length code word which corresponds image which relies upon such error concealment, 

to the run-level pair. If, on the other hand, the run-level pair Accordingly, it would be desirable to provide a method 

is not on the list, then the run-level pair is encoded as a 5 and apparatus for concealing errors where the visual effect 

predetermined symbol (such as an escape symbol) followed perceived by a viewer is negligible, and where the method 

by a fixed length codes to avoid long code words and to and apparatus adapt to the different types of available 

reduce the cost of implementation. information as may be available to provide such conceal- 

The run-length encoded and Huffman encoded output of ment. 

the variable length encoder 110 provides a coded video io 

bitstteam. Picture type determination circuit 112 determines SUMMARY OF THE INVENTION 

whether the frame being encoded is a P picture, an I picture It is an object of the invention to provide a method and 

or a B picture. In the case of a P or I picture, picture type apparatus for concealing errors during decoding of com- 

determination circuit 110 causes the motion vector generator pressed video signals. 

104 to generate an appropriate motion vector which is then 15 Jt j s a f^ber object of the invention to provide a method 

provided to variable length encoder 110. Such motion vector anc [ apparatus for detecting errors which do not produce 

is then coded and combined with the output of variable illegal syntax. 

length encoder 110. ft fa a fcatUfC of tfac invention to utilize a temporal 

Referring now to FIGS. 2 and 3, the concept of motion prediction of a motion vector t0 generate a macroblock 

compensation is explained. Motion compensation improves 20 which win effectivel roDaa i an crror ^ a data strcam . 

compression of P and B pictures by removing temporal , . , , c r , . . _ _ 

redundancies between pictures. With MPEG 2, it operates at 11 * a fealure of «^ "ivcntion to compare DC 

the macroblock level. For example, a previous frame 200 <?f™f» of a cu " en ' ™<™block to a predicted coeffi- 

i ti i >+n<% cient to determine whether an error which does not produce 

contains, among other macroblocks, a macroblock 202 con- ^ ^ ^ r 

sisting of 16 pixels (also referred to as "pels") by 16 lines. 25 e ^ a svntax ^ occurre 

Motion compensation relies on the fact that, except for scene 11 * aa advantage of the invention to improve the quality 

cuts, most images remain in the same location from frame to of concealment of an error in a data stream, 

frame, whereas others move only a short distance. Thus, It is a further advantage of the invention to improve the 

such motion can be described as a two-dimensional motion quality of detection of an error in a data stream, 

vector that specifies where to retrieve a macroblock from a 30 According to one aspect of the invention, an apparatus for 

previously decoded frame to thereby predict the pixel values concealing errors includes a detector for detecting the pres- 

of a current macroblock. Thus, a macroblock 300 of a ence of an error in data representing the current macroblock, 

current frame 302 can be represented by the macroblock 202 a system for estimating the at least one motion vector based 

(of FIG. 2) as modified by a two dimensional motion vector upon a difference between a forward reference frame at the 

304. It is to be understood that the macroblock 300 may or 35 current macroblock and a decoded motion vector for the 

may not be within the same boundaries surrounding mac- forward reference frame at the current macroblock, and a 

roblock 202 in the previous frame 200. system for estimating the current macroblock based upon the 

After a macroblock has been compressed using motion estimated at least one motion vector. According to another 

compensation, it contains both the prediction (commonly aspect of the invention, a method for concealing errors 

referred to as "motion vectors") and temporal difference 40 includes the steps of detecting the presence of an error in 

(commonly referred to as "error terms**) between the refer- data representing the current macroblock, estimating the at 

ence macroblock and the macroblock being coded. least one motion vector based upon a difference between a 

Returning to FIG. 1, when the coded video bitstream forward reference frame at the current macroblock and a 

output from variable length encoder 110 is recorded onto a decoded motion vector for the forward reference frame at 

recording medium such as an optical disk, and such recorded 45 the current macroblock, and estimating the current macrob- 

information is reproduced for local use, although not com- lock based upon the estimated at least one motion vector, 

pletely error free, the decoded (coded) video bit stream is, These and other objects, features and advantages will 

generally, sufficiently error free so as to not require addi- become apparent when considered with reference to the 

tional techniques to compensate for errors in the decoded following description and the accompanying drawings, 

video bit stream. Such a coded video bit stream is typically 50 

referred to as a "program stream." When the coded video BRIEF DESCRIPTION OF THE DRAWINGS 

bitstream output from variable length encoder 110 is trans- FIG. 1 is a simplified block diagram of a MPEG 2 video 

ported by, for example, satellite or cable transmission encoder. 

systems, either directly from variable length encoder 110 or FIG. 2 is an illustration of a macroblock within a previous 

from a recording medium onto which the coded video 55 ff ame 

bitstream has been recorded, the probability of errors in the „ T „ ~ . ... . f 

a a a u* ♦ — - _ * tiij. 3 is an illustration ot a macroblock within a current 

decoded video bitstream increases. Such a coded bitstream f rame 

is typically referred to as a "transport stream." 

Since traditional error detection and correction systems, FIG * 4 is simplified block diagram of a MPEG 2 video 

such as interleaving, require a significant amount of over- 60 decoder of the P reseat invention. 

head as well as a significant amount of data processing when FIG. 5 is a block diagram of a motion compensation 

decoding coded video bitstream signals, current video system of the present invention. 

decoding systems rely upon error concealment as opposed to FIG. 6 is a state diagram which illustrates reference block 

error correction. In contrast to error correction, which fetch control of the address generation and control unit of 

attempts to reconstruct lost or corrupt data, error conceal- 65 FIG. 5. 

ment aims to generate data which can be substituted for the FIG. 7 is a flow chart of a method for estimating mac- 
lost or corrupt data, where any discrepancies in image roblocks in accordance with the present invention. 
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FIG. 8 is a flow chart of a method for estimating motion In the preferred embodiment of the invention, in non- 
vectors in the temporal domain in accordance with the DVD operation, the sub picture/vertical blanking interval 
present invention. decoder 420 decodes and displays on screen display 
FIG. 9 is a flow chart of a method for estimating motion ("OSD") bitstreams. In DVD operation, however, OSD 
vectors in the spatial domain in accordance with the present 5 bitstreams are instead decoded by video post filter on screen 
invention. display system 426. 

FIG. 10 is a flow chart of a method for macroblock RISC CPU 408 operates to parse the video bitstream in 

estimation utilizing estimated motion vectors. order to control the decoder 400. RISC CPU 408 also 

FIG, 11 is a flow chart of a method for macroblock partially decodes the video bitstream (for example, decoding 

estimation without the use of estimated motion vectors. 30 of top-level data such as headers) and also controls various 

of the other units within decoder 400 through RBUS 404. A 

DETAILED DESCRIPTION OF THE portion of the parsing is also performed by sub picture/video 

PREFERRED EMBODIMENTS blanking interval decoder 420. In further detail, RISC CPU 

Referring now to FIG. 4, a block diagram of an MPEG 2 „ I 08 ca ° 1?"!^ to ^ han S e the P osition of f SP window 

decoder 400 is shown. Decoder 400 utilizes two internal 15 trough RBUS 404 Thus, a user can move the SP window 

busses, a GBUS 402 and an RBUS 404. In the preferred U P °j. down ihtov & a command to CPU 404 with a Y 

embodiment of the invention, GBUS 402 is a 64 bit bus coordinate as a parameter. 

which is utilized for data transfer between DRAM 406 and box mii 418 15 essentially a vertical decimation 

specific blocks of decoder 400 which are described below. In mter ^ downloadable coefficients. Letter box unit 418 

the preferred embodiment of the invention, DRAM 406 is a operates to decimate an active area of a frame which has a 

static dynamic random access memory, although other types ratio of 4:3 - Therefore, for PAL sequences, letter box unit 

ofmemories may be utilized. RBUS 404 is an 8 bit but used 418 converts a 720x576 frame to a 720x432 frame. For 

primarily for control of specific blocks through reduced NTSC sequences, letter box unit 418 converts a 720x480 

instruction set computing ("RISC") CPU 408. RISC CPU frame 10 a 720x360 frame. However, in both cases, the 

408, which is coupled to both GBUS 402 and RBUS 404, 25 active picture area is centered with respect to a display area, 

operates to control the functionality of specific blocks, as Host processor 434 and RISC CPU 408 utilize DRAM 

more particularly described below, as well as performing a 406 to exchange messages, commands and status informa- 

portion of video bitstream decoding. tion. In the preferred embodiment of the invention, proces- 

Decoder 400 includes a demultiplexer 410 which is 30 sor 434 and CPU 408 nave tne capability to interrupt each 

coupled to both GBUS 402 and RBUS 404. Similarly, a other - In operation, CPU 408 provides a host command 

video decoder 412, an audio decoder 414, a host interface P arser t0 execute such commands from host processor 434. 

416, a letter box unit 418, and a sub picture/vertical blanking A l yP ical sequence of events during execution of a command 

interval decoder 420 are each coupled to both GBUS 402 bv host processor 434 is: 

and RBUS 404. An RBUS controller 422, an NTSC/PAL 35 1. Host processor 434 writes a command to DRAM 406 

encoder 424, a video post filter/on screen display system 426 and interrupts CPU 408. 

and an audio clock generator 428 are each coupled to RBUS 2. CPU 408 reads the command and parameters from 

404. Audio clock generator 428 outputs a clock signal DRAM 406. 

ACLK. A memory controller 430 is coupled to GBUS 402. 3. CPU 408 acknowledges the command by writing a 

A clock generator 432, which provides a clock signal 40 status variable to DRAM 406. 

SCLK, is coupled to host interface 416. An output of letter 4. Command parser of CPU 408 parses the command and 

box unit 418 is provided to video post filter on screen display executes it. 

system 426. Sub picture/vertical blanking interval decoder 5. Optionally, CPU 408 interrupts host processor 434 

420 is coupled to video post filter on screen display system upon completion of the command to report status. 

426, which system provides its output to NTSC/PAL 45 Alternatively, CPU 408 polls a DRAM command buffer (not 

encoder 424. Sub picture/vertical blanking interval decoder shown) for every field sync. This buffer is a ring buffer 

420 is coupled to video post filter on screen display system where a write pointer is maintained by host processor 434 

426. A host processor 434 interfaces with host interface 416. while a read pointer is maintained by CPU 408. 

In the preferred embodiment of the invention, when Video decoder 412 contains an inverse cosine discrete 
utilized for DVD applications, sub picture/vertical blanking 50 transformer, a variable length decoder 436, a motion corn- 
interval decoder 420 and letter box unit 418 are hardwired pensation unit 438 and an inverse discrete cosine trans- 
units. Letter box unit 418 performs a 4-tap vertical filtering former 440. Video decoder 412 decodes a coded video data 
and sub-sampling of a video bit stream provided through stream received through GBUS 402 and provides a decoded 
GBUS 402 and operates to control the video post filter/on stream to NTSC/PAL encoder 424 through RBUS 404. 
screen display system 426. Sub picture/vertical blanking 55 NTSC/PAL encoder converts the decoded stream into an 
interval decoder 420 operates to decode sub picture ("SP") analog signal suitable for display on a television monitor 
and vertical blanking interval ("VBI") information in the having NTSC and/or PAL signal inputs, 
video bit stream. Typically, a sub picture bitstream consists Demultiplexer 410 operates on data entering decoder 400. 
of subtitles or menu items. For example, this would include In particular, such data is in the form of packets, and 
karaoke and menu highlighting. Since the VBI bitstream and 60 includes audio, video and other streams of multiplexed 
the SP bitstream are very similar in syntax and functionality packets. Demultiplexer 410 selects desired audio packets, 
(under MPEG 2 specifications), the functionality for decod- video packets and other desired information packets, but 
ing both types of bitstreams is incorporated into a single sub rejects the other packets within the video bitstream. For 
picture/vertical blanking interval decoder 420. Thus, in the example, audio packets representing audio in several differ- 
preferred embodiment of the invention, decoding of the VBI 65 ent languages may be present in the video bitstream. Based 
bit stream occurs during the vertical blanking period, while upon commands from host processor 434, demultiplexer 410 
SP bitstream decoding occurs during active display periods. selects only those audio packets corresponding to that lan- 
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guage which is selected for presentation with the corre- 
sponding video packets. 

Host interface 416 provides a glueless interface for host 
processor 434. RBUS controller 422 sends out messages on 
RBUS 404 and acts as an arbitrator for RBUS 404. Clock 
generator 432 provides a clock signal SCLK to the various 
units within decoder 400, while audio clock generator 428 
provides a clock signal ACLK to a digital-to-analog con- 
venor (not shown) which through GBUS 402 receives 
digital audio signals from audio decoder 414. These digital 
audio signals are converted to analog signals which can be 
perceived by humans. 

With reference now to FIG. 5, the motion compensation 
system of the video decoder 412 of FIG. 4 is now explained. 
The motion compensation unit 500 includes an address 
generation and control unit 502. The address generation and 
control unit 502 corresponds to the memory controller 430 
of FIG. 4. The address generation and control unit 502 
accepts motion vectors from variable length decoder 436 
and calculates a starting address of a reference macroblock. 
Based upon such calculation, the address generation and 
control unit 502 issues a data transfer request to the memory 
controller unit 430. In the preferred embodiment of the 
invention, data transfer occurs in 64 bit (8 byte) segments at 
addresses aligned at 8-byte boundaries. When this data 
returns from the DRAM 406, the data are latched within the 
motion compensation unit 500, Each 8 bit element of these 
latched data is then run through horizontal and vertical 
half-pel filters 504, and the resulting data is stored in the 
prediction RAM (random access memory) 506. 

Since the starting address of the reference block may not 
be aligned at the 8-byte aligned address, multiplexing is 
required at the input of the prediction RAM 506. For 
I -pictures, prediction is not required and, therefore, the 
motion compensation unit sits idle. For both P and 
B -pictures, however, prediction data is required for recon- 
struction by the reconstruction unit 508 of decoded picture 
data. In the case of B-pictures, the predicted data is obtained 
by averaging two such predictions, that is, the output of the 
half-pel filters at the time and a value from a prediction 
RAM 506 that was stored after a forward prediction. The 
reconstruction unit 508 supports this averaging of the half- 
pel filters 504. 

An estimation RAM 510 holds coefficient data trans- 
formed in the inverse discrete cosine transformer 440, 
Reconstruction of each picture starts once the estimation 
RAM 510 is full. The motion compensation unit 500, at such 
a time, issues a data transfer request and begins reconstruc- 
tion. The reconstruction basically consists of adding signed 
numbers from the output of the inverse discrete cosine 
transformer stored in the estimation RAM 510 to the outputs 
(stored in the prediction RAM 506) of the half-pel filters 504 
for non-intra blocks. For intra-blocks however, the addition 
is not required. In such a case, the adder output is clipped 
before it is latched at the output of the reconstruction unit 
508 when reconstruction of the picture occurs. 

Referring now to the state diagram of FIG. 6, a state 
machine 600 which represents the functionality of the 
address generation and control unit 502 in regard to the 
transfer of reference picture data from DRAM 406 and the 
construction of a macroblock is now explained. From a start 
state 602, a state machine 600 proceeds to a get address state 
604. If no motion compensation is to be used to construct the 
macroblock, state machine 600 proceeds to a y0_wait state 
606. If only backward motion compensation is to be utilized, 
then state machine 600 proceeds to state 608 to get or fetch 
a previous macroblock b which will serve as a reference 
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macroblock. If however forward motion compensation is to 
be utilized, then state machine 600 proceeds to state 610 to 
get or fetch a forward macroblock f which will serve as a 
reference macroblock. State machine 600 then proceeds to 

5 the y(L_wait state 606. If the macroblock to be constructed 
is to be based upon both the forward macroblock f and the 
previous macroblock b f then state machine 600 proceeds 
from state 610 to state 608 to also get or fetch a previous 
macroblock. In such an instance, both the forward and the 
previous macroblock will serve as reference macroblocks. 

At state 606, state machine 600 waits for luminance data 
to be received in regard to the reference macroblock or 
macroblocks. At state 612 the luminance portion of the 
macroblock to be constructed is reconstructed. At state 614 
state machine waits for chrominance data to be received in 

15 regard to the reference macroblock or macroblocks. At state 
618 reconstruction of the chrominance portion of the mac- 
roblock to be constructed occurs. Upon completion of 
chrominance data reconstruction, state machine 600 pro- 
ceeds to state 620 to await an instruction to construct a new 

20 macroblock. 

Similar to the case of the previously constructed 
macroblock, state machine 600 then proceeds to a get 
address 1 state 622. if no motion compensation is to be used 
to construct the macroblock, state machine 600 proceeds to 

25 a yl_wait state 624. If only backward motion compensation 
is to be utilized, then state machine 600 proceeds to state 626 
to get or fetch a previous macroblock bl which will serve as 
a reference macroblock. If, however, forward motion com- 
pensation is to be utilized, then state machine 600 proceeds 

30 to state 628 to get or fetch a forward macroblock fl which 
will serve as a reference macroblock. State machine 600 
then proceeds to the yl_wait state 624. If the new macrob- 
lock to be constructed is to be based upon both the forward 
macroblock fl and the previous macroblock bl, state 

35 machine 600 proceeds from state 628 to state 626 to also get 
or fetch the previous macroblock bl. In such an instance 
both the forward macroblock fl and the previous macrob- 
lock bl will serve as reference macroblocks. 
At state 624, state machine 600 waits for luminance data 

40 to be received in regard to the reference macroblock or 
macroblocks. At state 630 the luminance portion of the 
macroblock to be constructed is reconstructed. At state 632 
state machine waits for chrominance data to be received in 
regard to the reference macroblock or macroblocks. At state 

45 634 reconstruction of the chrominance portion of the mac- 
roblock to be constructed occurs. Upon completion of such 
chrominance data reconstruction, state machine 600 pro- 
ceeds to back to start state 602. 
As the state transitions of FIG. 6 depict, once an address 

50 is obtained as in state 604 or 622, sampling occurs to 
determine whether or not motion compensation is required 
for a macroblock. For coded pictures that require motion 
compensated reference calculations, state machine 600 waits 
until the motion vector FIFO memory of the variable length 

55 decoder 436 is not empty. The address generation and 
control unit 502 then generates a request for a motion vector. 
Two consecutive requests, one for X (horizontal) and one for 
Y (vertical) components of the motion vectors are made. 
Once the address generation and control unit 502 obtains 

60 both components of the motion vector, the address of the 
reference block is calculated. The address generation and 
control unit 502 then sends a request for data transfer to the 
memory controller unit. 
As mentioned above, when a motion vector points to a 

65 sub-pixel location instead of to an exact pixel location, in 
order to more accurately represent a P or B picture, it is 
necessary to generate half -pixel (half-pel) data. 
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In the event that during transport (or even during local 
reproduction) of a recorded coded video bit stream, errors 
are detected in the video bit stream in regard to a particular 
macroblock. In the preferred embodiment of the invention, 
the smallest unit of concealment is a slice. A slice consists 5 
of a series of sequential macroblocks. In order to provide 
such concealment, motion vectors are estimated using either 
temporal prediction or spatial prediction. In the case of 
spatial prediction, pixels from a successfully decoded mac- 
roblock are copied for use in decoding the macroblock 
having a data error. In the case of temporal prediction, 
motion vectors from a successfully decoded macroblock are 
utilized to predict a new motion vector field in order to 
decode the macroblock having a data error. 

In further detail, if in decoding a coded video bitstream, 
a frame K is missing a macroblock or a portion of a 15 
macroblock (as could result from a data error), the basic 
concept is that if there is a motion of an object from a frame 
K-2 (that is, two frames prior to frame K), one can assume 
that this motion will most likely continue from frame K-2 
up through frame K. Therefore, the assumption is that the 20 
motion will be basically linear. Based upon that assumption, 
the present invention estimates pixels and motion vectors, 
the estimation method depending upon the data available for 
such estimation. 

In utilizing the estimated pixels and/or motion vectors, 25 
actual concealment is delayed until the occurrence of a later 
slice. When an error is detected, such error is recorded 
together with its location. In particular, data regarding the 
error and its location is written into a register and, after a 
second or third subsequent slice, an interrupt command is 30 
issued to allow processing of the macroblock utilizing 
estimated pixels and/or motion vectors. In the preferred 
embodiment of the invention, when applied to interlaced 
video, although four motion vectors per macroblock are 
available for use, only two motion vectors are utilized, as 35 
will be detailed below. 

Referring now to FIG. 7, the error concealment of the 
present invention is explained in further detail. Once an error 
has been detected and an interrupt command has been 
issued, the error concealment algorithm starts at step 700. 40 
The motion compensation unit 438 first attempts to estimate 
motion vectors in the temporal domain at step 702. FIG. 8 
illustrates such a method. The algorithm starts at step 800. 
At step 802, the motion compensation unit 438 determines 
whether a decoded motion vector for a forward reference 45 
frame at the macroblock positioned by a vector p is avail- 
able. This motion vector is designated as MV(k-m,lT), 
where k is the current frame, and m is the frame index 
difference between the current frame and a forward refer- 5Q 
ence frame. If not, motion vector estimation in the temporal 
domain is not performed and the algorithm proceeds to step 
804, which indicates a failed attempt. If a decoded motion 
vector for a forward reference frame at the macroblock 
positioned by the vector p is available, the algorithm pro- 55 
ceeds to step 806, which determines whether a decoded 
motion vector is available for the difference between (1) a 
forward reference frame at the macroblock positioned by the 
vector p; and (2) a decoded motion vector for a forward 
reference frame at the macroblock positioned by the vector gQ 
p, where such decoded motion vector is designated by 

MV(k-m, p ). If not available, the algorithm proceeds to 
step 804 to indicate a failed attempt. If available, the 
algorithm proceeds to step 808 at which an estimated motion 
vector for a current frame, the k-th frame, at the macroblock 65 
positioned by the vector p is determined. Such estimated 
motion vector is taken to be equal to the difference between 



(1) a forward reference frame at the macroblock positioned 
by the vector p; and (2) a decoded motion vector for a 
forward reference frame at the macroblock positioned by the 
vector p. The algorithm then proceeds to step 810 which 
indicates a successful motion vector estimation in the tem- 
poral domain. 

Returning to FIG, 7, at step 704, it is determined whether 
motion vector estimation in the temporal domain was suc- 
cessful. If so, the algorithm proceeds to step 706, where 
based upon the estimated motion vector, the motion vector 
to be used for estimating the subject macroblock is updated. 
If the motion vector estimation in the temporal domain was 
not successful, the algorithm proceeds to step 708, where 
motion vector estimation is performed in the spatial domain. 
The algorithm for such estimation is shown in FIG. 9. The 
algorithm starts at step 900 of FIG. 9 and proceeds to step 
902, where it is determined whether a decoded motion 
vector for the macroblock located immediately above the 
estimating macroblock is available. Such a motion vector is 

designated by MV(k, p -(1,0)). If not, a failure is indicated 
at step 904. If a decoded motion vector for the macroblock 
located immediately above the estimating macroblock is 
available, then, at step 906, the motion vector for the current 
frame, the k-th frame at the macroblock positioned by the 
vector p, ~MV(k,^), is estimated to be equal to the decoded 
motion vector, the macroblock located immediately above 

the estimating macroblock, MV(k, p -(1,0)), where (1,0) is 
a vector indicating a row index as 1 and a column index as 
0. The algorithm then proceeds to step 908 which indicates 
a successive motion vector estimation in the spatial domain. 

Returning again to FIG. 7, at step 710, a determination is 
made whether or not a motion vector estimation has been 
successfully made in the spatial domain. If such a determi- 
nation has been made (step 908), then the motion vector for 
the current macroblock is updated at step 706. Then, at step 
712, the current macroblock is estimated using the just 
estimated motion vector, whether that motion vector is 
estimated in the temporal domain at step 702 or, in the 
spatial domain, at step 708. 

As shown in FIG. 10, at step 1000, macroblock estimation 
with the estimated motion vector is started. At step 1002, the 
estimated macroblock for the current frame, the k-th frame, 
at the macroblock positioned by the estimated motion vector 

p, ~MB(k, p), is estimated to be equal to the decoded 
macroblock of the difference of (1) the forward reference 
frame at the macroblock positioned by the vector p; and (2) 
the estimated motion vector for the current frame, the k-th 
frame, at the macroblock positioned by the vector p. This 

decoded macroblock is designated as MB(k-m, p — MV(k, 

p)), where m is the frame index difference between the 
current frame and a forward reference frame. Upon comple- 
tion of estimation of the current macroblock at step 712, the 
algorithm is completed at step 714. 

Returning to step 710 of FIG. 7, if motion vector estima- 
tion in the spatial domain is not successful, then, at step 716, 
the current macroblock is estimated without the use of an 
estimated motion vector. Step 716 is detailed in FIG. 11. In 
FIG. 11, macroblock estimation without use of an estimated 
motion vector starts at step 1100. At step 1102, it is deter- 
mined whether the macroblock for the frame preceding the 
current frame (the k-th frame being the current frame) 

positioned by the vector p, MB(k,-l,~p) is available. If such 
macroblock is available, then, at step 1104, the current 
macroblock positioned by the vector p is estimated to be 
equal to the macroblock for the frame preceding the current 
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frame positioned by the vector p. The algorithm is then 
completed as indicated at step 714. 

If the macroblock for the frame preceding the current 
frame (the k-th frame being the current frame) positioned by 
the vector p is not available, then, at step 1106, it is 
determined whether the macroblock for the current frame 
positioned by the vector p but indexed by minus 1 row and 

in the same column, MB(k, p -(1,0) is available, where (1,0) 
is a vector indicating a row index as 1 and a column index 
as 0. If such a macroblock is available, then, at step 1108, the 
current macroblock (for the current frame, the k-th frame, 
positioned by the vector p) is estimated to be equal to the 
macroblock for the current frame positioned by the vector p 
but indexed by minus 1 row and in the same column, MP(k, 

p -(1,0)). The algorithm is then completed as indicated at 
step 714. 

If it is determined that the macroblock for the current 
frame positioned by the vector p but indexed by minus 1 row 
and in the same column, is not available, then, at step 1U0, 
it is then determined whether the decoded macroblock for 
the macroblock located immediately above the macroblock 
to be estimated, MB(k,p+(l,0)), is available, where (1,0) is 
a vector indicating a row index as 1 and a column index as 
0. If the decoded macroblock for the macroblock located 
immediately above the macroblock to be estimated is 
available, then, at step 1112, the estimated macroblock for 
the current frame, the k-th frame, at the macroblock posi- 
tioned by the vector p is estimated to be equal to such 
decoded macroblock for the macroblock located immedi- 
ately above the macroblock to be estimated, MB(k, 

p +(1,0)). The algorithm is then completed as indicated at 
step 714. In the event that the decoded macroblock for the 
macroblock located immediately above the macroblock to 
be estimated is not available, then the macroblock estima- 
tion without an estimated motion vector fails, as indicated at 
step 1114. In this case, the macroblock can be left blank. 

It is to be understood that, although the present invention 
has been described in relation to decoding of a coded video 
bit stream, the present invention is also applicable to the 40 
coding of a video bit stream, where an error is detected 
during or after coding and the error is concealed prior to 
recording or transport. 

Although only certain embodiments have been described 
in detail, those having ordinary skill in the art will certainly 45 
understand that many modifications are possible without 
departing from the teachings hereof. All such modifications 
are intended to be encompassed within the following claims. 

We claim: 

1. A method of error concealment in decoding a coded 
video bit stream, comprising the steps of: 

estimating a motion vector in a temporal domain, the step 

of estimating the motion vector in the temporal domain 

including the steps of: 

detecting a first decoded motion vector for a forward 
reference frame at a macroblock positioned by a 
vector; 

detecting a second decoded motion vector for the 
difference between the forward reference frame at 
the macroblock positioned by the vector and the 
detected decoded motion vector for the forward 
reference frame at the macroblock positioned by the 
vector; and 

determining in a temporal domain an estimated motion 
vector for a current frame at the macroblock posi- 
tioned by the vector based upon the detected second 
decoded motion vector: 
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if the step of estimating the motion vector in the temporal 
domain is unsuccessful, estimating the motion vector in 
a spatial domain; 

updating a motion vector for use in estimating the mac- 
roblock based on the estimated motion vector deter- 
mined in either the temporal or the spatial domain; and 

estimating the macroblock based on the estimated motion 
vector. 

2. The method of claim 1 wherein the estimated motion 
vector for the current frame determined in the temporal 
domain equals the detected second decoded motion vector. 

3. The method of claim 2 wherein the forward reference 
frame and a frame index comprise the current frame. 

4. The method of claim 1 wherein the forward reference 
frame and a frame index comprise the current frame. 

5. The method of claim 1 wherein the step of estimating 
the motion vector in the spatial domain includes the steps of: 

detecting a third decoded motion vector for the current 
frame at a macroblock positioned immediately above 
the current macroblock; and 

determining in a spatial domain the estimated motion 
vector for the current frame at the macroblock posi- 
tioned by the vector based upon the detected third 
decoded motion vector. 

6. The method of claim 5 wherein the macroblock posi- 
tioned immediately above the current macroblock is defined 
by a vector having a row index of 1 and a column index of 
0. 

7. An apparatus for error concealment in decoding a coded 
video bit stream, comprising: 

means for estimating a motion vector in a temporal 
domain, the means for estimating the motion vector in 
the temporal domain including: 
means for detecting a first decoded motion vector for a 

forward reference frame at a macroblock positioned 

by a vector; 

means for detecting a second decoded motion vector 
for the difference between the forward reference 
frame at the macroblock positioned by the vector and 
the detected decoded motion vector for the forward 
reference frame at the macroblock positioned by the 
vector; and 

means for determining in a temporal domain an esti- 
mated motion vector for a current frame at the 
macroblock positioned by the vector based upon the 
detected second decoded motion vector; 
means for estimating a motion vector in a spatial domain, 
wherein when the means for estimating the motion 
vector in the temporal domain is unsuccessful, the 
means for estimating the motion vector in the spatial 
domain determines the estimated motion vector; 
means for updating a motion vector for use in estimating 
the macroblock based on the estimated motion vector 
determined in either the temporal or the spatial domain; 
and 

means for estimating the macroblock based on the esti- 
mated motion vector. 

8. The apparatus of claim 7 wherein the estimated motion 
vector for the current frame in the temporal domain equals 
the detected second decoded motion vector. 

9. The apparatus of claim 8 wherein the forward reference 
frame and a frame index comprise the current frame. 

10. The apparatus of claim 7 wherein the forward refer- 
ence frame and a frame index comprise the current frame. 

11. The apparatus of claim 7 wherein said means for 
estimating said motion vector in said spatial domain 
includes: 
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means for detecting a third decoded motion vector for the 
current frame at a macroblock positioned immediately 
above the current macroblock; and 

means determining in a spatial domain the estimated 
motion vector for the current frame at the macroblock 
positioned by the vector based upon the detected third 
decoded motion vector. 

12. The apparatus of claim 11 wherein the macroblock 
positioned immediately above the current macroblock is 
defined by a vector having a row index of 1 and a column 
index of 0. 

13. An apparatus for error concealment in decoding a 
coded video bit stream, comprising: 

a first motion vector estimating unit for estimating a 

motion vector in a temporal domain, the first motion 

vector estimating unit including: 

a first detection unit for detecting a first decoded 
motion vector for a forward reference frame at a 
macroblock positioned by a vector; 

a second detection unit for detecting a second decoded 
motion vector for the difference between the forward 
reference frame at the macroblock positioned by the 
vector and the detected decoded motion vector for 
the forward reference frame at the macroblock posi- 
tioned by the vector; and 

a first determining unit for determining in a temporal 
domain an estimated motion vector for a current 
frame at the macroblock positioned by the vector 
based upon the detected second decoded motion 
vector; 

a second motion vector estimating unit for estimating a 
motion vector in a spatial domain, wherein when the 
second motion vector estimating unit is configured to 
estimate the motion vector in the spatial domain when 
the first motion vector estimating unit is unsuccessful in 
estimating the motion vector in a temporal domain; 

an updating unit for updating a motion vector for use in 
estimating the macroblock based on the estimated 
motion vector determined in either the temporal or the 
spatial domain; and 

an estimating unit for estimating the macroblock based on 
the estimated motion vector. 

14. The apparatus of claim 13 wherein the estimated 
motion vector for the current frame in the temporal domain 
equals the detected second decoded motion vector. 
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15. The apparatus of claim 14 wherein the forward 
reference frame and a frame index comprise the current 
frame. 

16. The apparatus of claim 15 wherein said estimating 
unit includes: 

a third detection unit for detecting a third decoded motion 
vector for the current frame at a macroblock positioned 
immediately above the current macroblock; and 

a second determining unit for determining in a spatial 
domain the estimated motion vector for the current 
frame at the macroblock positioned by the vector based 
upon the detected third decoded motion vector. 

17. The apparatus of claim 16 wherein the macroblock 
positioned immediately above the current macroblock is 
defined by a vector having a row index of 1 and a column 
index of 0. 

18. The apparatus of claim 13 wherein the forward 
reference frame and a frame index comprise the current 
frame. 

19. The apparatus of claim 18 wherein said estimating 
unit includes: 

a third detection unit for detecting a third decoded motion 
vector for the current frame at a macroblock positioned 
immediately above the current macroblock; and 

a second determining unit for determining in a spatial 
domain the estimated motion vector for the current 
frame at the macroblock positioned by the vector based 
upon the detected third decoded motion vector. 

20. The apparatus of claim 19 wherein the macroblock 
positioned immediately above the current macroblock is 
defined by a vector having a row index of 1 and a column 
index of 0. 

21. The apparatus of claim 13 wherein said estimating 
unit includes: 

a third detection unit for detecting a third decoded motion 
vector for the current frame at a macroblock positioned 
immediately above the current macroblock; and 

a second determining unit for determining in a spatial 
domain the estimated motion vector for the current 
frame at the macroblock positioned by the vector based 
upon the detected third decoded motion vector. 

22. The apparatus of claim 21 wherein the macroblock 
positioned immediately above the current macroblock is 
defined by a vector having a row index of 1 and a column 
index of 0. 
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